{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {},
  "cells": [
    {
      "metadata": {},
      "source": [
        "<td>\n",
        "   <a target=\"_blank\" href=\"https://labelbox.com\" ><img src=\"https://labelbox.com/blog/content/images/2021/02/logo-v4.svg\" width=256/></a>\n",
        "</td>"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "<td>\n",
        "<a href=\"https://colab.research.google.com/github/Labelbox/labelbox-python/blob/master/examples/prediction_upload/text_predictions.ipynb\" target=\"_blank\"><img\n",
        "src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"></a>\n",
        "</td>\n",
        "\n",
        "<td>\n",
        "<a href=\"https://github.com/Labelbox/labelbox-python/blob/master/examples/prediction_upload/text_predictions.ipynb\" target=\"_blank\"><img\n",
        "src=\"https://img.shields.io/badge/GitHub-100000?logo=github&logoColor=white\" alt=\"GitHub\"></a>\n",
        "</td>"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "# Text Prediction Import\n",
        "* This notebook walks you through the process of uploading model predictions to a Model Run. This notebook provides an example for each supported prediction type for text assets. \n",
        "\n",
        "Supported annotations that can be uploaded through the SDK: \n",
        "\n",
        "* Entity\n",
        "* Classification radio \n",
        "* Classification checklist \n",
        "* Classification free-form text \n",
        "\n",
        "**Not** supported:\n",
        "* Segmentation mask\n",
        "* Polygon\n",
        "* Bounding box \n",
        "* Polyline\n",
        "* Point \n",
        "\n",
        "\n",
        "A Model Run is a container for the predictions, annotations and metrics of a specific experiment in your ML model development cycle.\n",
        "\n"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "## Setup"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "!pip install -q \"labelbox[data]\""
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "import labelbox as lb\n",
        "import labelbox.types as lb_types\n",
        "import uuid"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "## Replace with your API Key \n",
        "Guides on [Create an API key](https://docs.labelbox.com/docs/create-an-api-key)"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "API_KEY= \"\"\n",
        "client = lb.Client(API_KEY)"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "## Supported Predictions"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "########## Entities ##########\n",
        "\n",
        "# Python annotation\n",
        "named_entity = lb_types.TextEntity(start=10, end=20)\n",
        "entities_prediction = lb_types.ObjectAnnotation(value=named_entity, name = \"named_entity\", confidence=0.5)\n",
        "\n",
        "\n",
        "# NDJSON\n",
        "entities_prediction_ndjson = { \n",
        "    \"name\": \"named_entity\",\n",
        "    \"confidence\": 0.5, \n",
        "    \"location\": { \n",
        "        \"start\": 10, \n",
        "        \"end\": 20 \n",
        "    }\n",
        "}"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "########## Classification - Radio (single choice ) ##########\n",
        "\n",
        "# Python annotation \n",
        "radio_prediction = lb_types.ClassificationAnnotation(\n",
        "    name=\"radio_question\",\n",
        "    value=lb_types.Radio(answer =\n",
        "        lb_types.ClassificationAnswer(name = \"first_radio_answer\", confidence=0.5)\n",
        "    )\n",
        ")\n",
        "\n",
        "# NDJSON\n",
        "radio_prediction_ndjson = {\n",
        "  \"name\": \"radio_question\",\n",
        "  \"answer\": {\"name\": \"first_radio_answer\", \"confidence\": 0.5}\n",
        "} "
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "########## Classification - Radio and Checklist (with subclassifcations)  ##########\n",
        "\n",
        "nested_radio_prediction = lb_types.ClassificationAnnotation(\n",
        "  name=\"nested_radio_question\",\n",
        "  value=lb_types.Radio(\n",
        "    answer=lb_types.ClassificationAnswer(\n",
        "      name=\"first_radio_answer\",\n",
        "      confidence=0.5, # Confidence scores should be added to the answer \n",
        "      classifications=[\n",
        "        lb_types.ClassificationAnnotation(\n",
        "          name=\"sub_radio_question\",\n",
        "          value=lb_types.Radio(\n",
        "            answer=lb_types.ClassificationAnswer(\n",
        "              name=\"first_sub_radio_answer\",\n",
        "              confidence=0.5 \n",
        "            )\n",
        "          )\n",
        "        )\n",
        "      ]\n",
        "    )\n",
        "  )\n",
        ")\n",
        "\n",
        "nested_radio_prediction_ndjson= {\n",
        "  \"name\": \"nested_radio_question\",\n",
        "  \"answer\": {\n",
        "      \"name\": \"first_radio_answer\",\n",
        "      \"confidence\": 0.5, # Confidence scores should be added to the answer \n",
        "      \"classifications\": [{\n",
        "          \"name\":\"sub_radio_question\",\n",
        "          \"answer\": { \"name\" : \"first_sub_radio_answer\", \n",
        "                     \"confidence\": 0.5}\n",
        "        }]\n",
        "    }\n",
        "}\n",
        "\n",
        "nested_checklist_prediction = lb_types.ClassificationAnnotation(\n",
        "  name=\"nested_checklist_question\",\n",
        "  value=lb_types.Checklist(\n",
        "    answer=[lb_types.ClassificationAnswer(\n",
        "      name=\"first_checklist_answer\",\n",
        "      confidence=0.5, # Confidence scores should be added to the answer \n",
        "      classifications=[\n",
        "        lb_types.ClassificationAnnotation(\n",
        "          name=\"sub_checklist_question\",\n",
        "          value=lb_types.Checklist(\n",
        "            answer=[lb_types.ClassificationAnswer(\n",
        "            name=\"first_sub_checklist_answer\",\n",
        "            confidence=0.5,\n",
        "          )]\n",
        "        ))\n",
        "      ]\n",
        "    )]\n",
        "  )\n",
        ")\n",
        "\n",
        "nested_checklist_prediction_ndjson = {\n",
        "  \"name\": \"nested_checklist_question\",\n",
        "  \"answer\": [{\n",
        "      \"name\": \"first_checklist_answer\", \n",
        "      \"confidence\": 0.5, # Confidence scores should be added to the answer \n",
        "      \"classifications\" : [\n",
        "        {\n",
        "          \"name\": \"sub_checklist_question\", \n",
        "          \"answer\": {\"name\": \"first_sub_checklist_answer\", \n",
        "                     \"confidence\": 0.5}\n",
        "        }          \n",
        "      ]         \n",
        "  }]\n",
        "}"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "########## Checklist ##########\n",
        "\n",
        "# Python annotation\n",
        "checklist_prediction = lb_types.ClassificationAnnotation(\n",
        "    name=\"checklist_question\",\n",
        "    value=lb_types.Checklist(\n",
        "        answer = [\n",
        "            lb_types.ClassificationAnswer(\n",
        "                name = \"first_checklist_answer\",\n",
        "                confidence=0.5\n",
        "            ),\n",
        "            lb_types.ClassificationAnswer(\n",
        "                name = \"second_checklist_answer\", \n",
        "                confidence=0.5\n",
        "            ),\n",
        "            lb_types.ClassificationAnswer(\n",
        "                name = \"third_checklist_answer\", \n",
        "                confidence=0.5\n",
        "            )\n",
        "    ])\n",
        "  )\n",
        "\n",
        "\n",
        "# NDJSON\n",
        "checklist_prediction_ndjson = {\n",
        "  \"name\": \"checklist_question\",\n",
        "  \"answer\": [\n",
        "    {\"name\": \"first_checklist_answer\", \"confidence\": 0.5}\n",
        "  ]\n",
        "}"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "########## Classification Free-Form text  ##########\n",
        "\n",
        "# Python annotation\n",
        "text_prediction = lb_types.ClassificationAnnotation(\n",
        "    name = \"free_text\", \n",
        "    value = lb_types.Text(answer=\"sample text\", confidence=0.5)\n",
        ")\n",
        "\n",
        "#  NDJSON\n",
        "text_prediction_ndjson = {\n",
        "  \"name\": \"free_text\",\n",
        "  \"answer\": \"sample text\",\n",
        "  \"confidence\": 0.5\n",
        "}"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "## Step 1: Import data rows into Catalog"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "# send a sample image as batch to the project\n",
        "global_key = \"lorem-ipsum.txt\"\n",
        "test_img_url = {\n",
        "    \"row_data\": \"https://storage.googleapis.com/labelbox-sample-datasets/nlp/lorem-ipsum.txt\",\n",
        "    \"global_key\": global_key\n",
        "}\n",
        "dataset = client.create_dataset(\n",
        "    name=\"text prediction demo dataset\",\n",
        "    iam_integration=None # Removing this argument will default to the organziation's default iam integration\n",
        "    )\n",
        "task = dataset.create_data_rows([test_img_url])\n",
        "task.wait_till_done()\n",
        "print(\"Errors:\",task.errors)\n",
        "print(\"Failed data rows:\", task.failed_data_rows)"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "## Step 2: Create/select an Ontology for your model predictions\n",
        "Your project should have the correct ontology setup with all the tools and classifications supported for your annotations, and the tool names and classification instructions should match the name/instructions fields in your annotations to ensure the correct feature schemas are matched.\n"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "## Setup the ontology and link the tools created above.\n",
        "\n",
        "ontology_builder = lb.OntologyBuilder(\n",
        "  classifications=[ # List of Classification objects\n",
        "    lb.Classification(\n",
        "      class_type=lb.Classification.Type.RADIO,\n",
        "      name=\"radio_question\", \n",
        "      options=[lb.Option(value=\"first_radio_answer\")]\n",
        "    ),\n",
        "    lb.Classification(\n",
        "      class_type=lb.Classification.Type.RADIO,\n",
        "      name=\"nested_radio_question\", \n",
        "      options=[\n",
        "        lb.Option(value=\"first_radio_answer\",\n",
        "          options=[\n",
        "              lb.Classification(\n",
        "                class_type=lb.Classification.Type.RADIO,\n",
        "                name=\"sub_radio_question\",\n",
        "                options=[\n",
        "                  lb.Option(value=\"first_sub_radio_answer\")\n",
        "                ]\n",
        "            ),\n",
        "          ]\n",
        "        )\n",
        "      ],\n",
        "    ),\n",
        "    lb.Classification(\n",
        "      class_type=lb.Classification.Type.CHECKLIST,\n",
        "      name=\"checklist_question\", \n",
        "      options=[\n",
        "        lb.Option(value=\"first_checklist_answer\"),\n",
        "        lb.Option(value=\"second_checklist_answer\"),\n",
        "        lb.Option(value=\"third_checklist_answer\")\n",
        "      ]\n",
        "    ), \n",
        "     lb.Classification(\n",
        "      class_type=lb.Classification.Type.TEXT,\n",
        "      name=\"free_text\"\n",
        "    ),\n",
        "    lb.Classification(\n",
        "      class_type=lb.Classification.Type.CHECKLIST,\n",
        "      name=\"nested_checklist_question\",\n",
        "      options=[\n",
        "          lb.Option(\"first_checklist_answer\",\n",
        "            options=[\n",
        "              lb.Classification(\n",
        "                  class_type=lb.Classification.Type.CHECKLIST,\n",
        "                  name=\"sub_checklist_question\", \n",
        "                  options=[lb.Option(\"first_sub_checklist_answer\")]\n",
        "              )\n",
        "          ]\n",
        "        )\n",
        "      ]\n",
        "    )\n",
        "  ],\n",
        "  tools=[ # List of Tool objects\n",
        "         lb.Tool(tool=lb.Tool.Type.NER,\n",
        "              name=\"named_entity\")\n",
        "    ]\n",
        ")\n",
        "\n",
        "ontology = client.create_ontology(\"Ontology Text Predictions\", ontology_builder.asdict() , media_type=lb.MediaType.Text)\n"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "## Step 3: Create a Model and Model Run"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "# create Model\n",
        "model = client.create_model(name=\"text_model_run_\"+ str(uuid.uuid4()), \n",
        "                            ontology_id=ontology.uid)\n",
        "# create Model Run\n",
        "model_run = model.create_model_run(\"iteration 1\")"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "## Step 4: Send data rows to the Model Run"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "model_run.upsert_data_rows(global_keys=[global_key])"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "## Step 5. Create the predictions payload\n",
        "\n",
        "Create the prediction payload using the snippets of code in the **Supported Predcitions** section\n",
        "\n",
        "Labelbox support two formats for the annotations payload: NDJSON and Python Annotation types. Both are described below to compose your annotations into Labels attached to the data rows.\n",
        "\n",
        "The resulting label_ndjson should have exactly the same content for annotations that are supported by both"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "# Create a Label for predictions\n",
        "label_predictions = []\n",
        "label_predictions.append(\n",
        "  lb_types.Label(\n",
        "    data=lb_types.TextData(global_key=global_key),\n",
        "    annotations = [\n",
        "      entities_prediction, \n",
        "      nested_radio_prediction,\n",
        "      radio_prediction, \n",
        "      checklist_prediction,\n",
        "      nested_checklist_prediction,\n",
        "      text_prediction,\n",
        "    ]\n",
        "  )\n",
        ")"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "If using NDJSON: "
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "label_ndjson_predictions= []\n",
        "for annot in [\n",
        "    entities_prediction_ndjson, \n",
        "    radio_prediction_ndjson, \n",
        "    checklist_prediction_ndjson,\n",
        "    text_prediction_ndjson, \n",
        "    nested_radio_prediction_ndjson,\n",
        "    nested_checklist_prediction_ndjson\n",
        "  ]:\n",
        "  annot.update({\n",
        "      \"dataRow\": {\"globalKey\": global_key}\n",
        "  })\n",
        "  label_ndjson_predictions.append(annot)"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "## Step 6. Upload the predictions payload to the Model Run "
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "# Upload the prediction label to the Model Run\n",
        "upload_job_prediction = model_run.add_predictions(\n",
        "    name=\"prediction_upload_job\"+str(uuid.uuid4()),\n",
        "    predictions=label_predictions)\n",
        "\n",
        "# Errors will appear for annotation uploads that failed.\n",
        "print(\"Errors:\", upload_job_prediction.errors)\n",
        "print(\"Status of uploads: \", upload_job_prediction.statuses)"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "## Step 7: Send annotations to the Model Run \n",
        "To send annotations to a Model Run, we must first import them into a project, create a label payload and then send them to the Model Run."
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "##### 7.1. Create a labelbox project"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "# Create a Labelbox project\n",
        "project = client.create_project(name=\"Text Prediction Import Demo\",\n",
        "                                    media_type=lb.MediaType.Text)\n",
        "project.setup_editor(ontology)"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "##### 7.2. Create a batch to send to the project "
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "project.create_batch(\n",
        "  \"batch_text_prediction_demo\", # Each batch in a project must have a unique name\n",
        "  global_keys=[global_key], # Paginated collection of data row objects, list of data row ids or global keys\n",
        "  priority=5 # priority between 1(Highest) - 5(lowest)\n",
        ")"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "##### 7.3 Create the annotations payload"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "named_entity = lb_types.TextEntity(start=10, end=20)\n",
        "entities_annotation = lb_types.ObjectAnnotation(value=named_entity, name = \"named_entity\")\n",
        "\n",
        "\n",
        "radio_annotation = lb_types.ClassificationAnnotation(\n",
        "    name=\"radio_question\",\n",
        "    value=lb_types.Radio(answer =\n",
        "        lb_types.ClassificationAnswer(name = \"first_radio_answer\")\n",
        "    )\n",
        ")\n",
        "\n",
        "nested_radio_annotation = lb_types.ClassificationAnnotation(\n",
        "  name=\"nested_radio_question\",\n",
        "  value=lb_types.Radio(\n",
        "    answer=lb_types.ClassificationAnswer(\n",
        "      name=\"first_radio_answer\",\n",
        "      classifications=[\n",
        "        lb_types.ClassificationAnnotation(\n",
        "          name=\"sub_radio_question\",\n",
        "          value=lb_types.Radio(\n",
        "            answer=lb_types.ClassificationAnswer(\n",
        "              name=\"first_sub_radio_answer\"\n",
        "            )\n",
        "          )\n",
        "        )\n",
        "      ]\n",
        "    )\n",
        "  )\n",
        ")\n",
        "\n",
        "nested_checklist_annotation = lb_types.ClassificationAnnotation(\n",
        "  name=\"nested_checklist_question\",\n",
        "  value=lb_types.Checklist(\n",
        "    answer=[lb_types.ClassificationAnswer(\n",
        "      name=\"first_checklist_answer\",\n",
        "      classifications=[\n",
        "        lb_types.ClassificationAnnotation(\n",
        "          name=\"sub_checklist_question\",\n",
        "          value=lb_types.Checklist(\n",
        "            answer=[lb_types.ClassificationAnswer(\n",
        "            name=\"first_sub_checklist_answer\"\n",
        "          )]\n",
        "        ))\n",
        "      ]\n",
        "    )]\n",
        "  )\n",
        ")\n",
        "\n",
        "checklist_annotation = lb_types.ClassificationAnnotation(\n",
        "    name=\"checklist_question\",\n",
        "    value=lb_types.Checklist(\n",
        "        answer = [\n",
        "            lb_types.ClassificationAnswer(\n",
        "                name = \"first_checklist_answer\"\n",
        "            ),\n",
        "            lb_types.ClassificationAnswer(\n",
        "                name = \"second_checklist_answer\" \n",
        "            ),\n",
        "            lb_types.ClassificationAnswer(\n",
        "                name = \"third_checklist_answer\" \n",
        "            )\n",
        "    ])\n",
        "  )\n",
        "\n",
        "text_annotation = lb_types.ClassificationAnnotation(\n",
        "    name = \"free_text\", \n",
        "    value = lb_types.Text(answer=\"sample text\")\n",
        ")"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "##### 7.4. Create the label object"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "# Create a Label object by identifying the applicable data row in Labelbox and providing a list of annotations\n",
        "# Create a Label for predictions\n",
        "label = []\n",
        "label.append(\n",
        "  lb_types.Label(\n",
        "    data=lb_types.TextData(global_key=global_key),\n",
        "    annotations = [\n",
        "      entities_annotation, \n",
        "      nested_radio_annotation,\n",
        "      radio_annotation, \n",
        "      checklist_annotation,\n",
        "      nested_checklist_annotation,\n",
        "      text_annotation,\n",
        "    ]\n",
        "  )\n",
        ")"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "##### 7.5. Upload annotations to the project using Label Import"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "upload_job_annotation = lb.LabelImport.create_from_objects(\n",
        "    client = client,\n",
        "    project_id = project.uid,\n",
        "    name=\"text_label_import_job\"+ str(uuid.uuid4()),\n",
        "    labels=label)\n",
        "\n",
        "upload_job_annotation.wait_until_done()\n",
        "# Errors will appear for annotation uploads that failed.\n",
        "print(\"Errors:\", upload_job_annotation.errors)\n",
        "print(\"Status of uploads: \", upload_job_annotation.statuses)"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "##### 7.6 Send the annotations to the Model Run"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "# get the labels id from the project\n",
        "model_run.upsert_labels(project_id=project.uid)"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "## Optional deletions for cleanup \n"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "# project.delete()\n",
        "# dataset.delete()"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    }
  ]
}